Face Recognition

Tina Rezvanian

1. Do existing pretrained models correctly recognize faces in images?

We will use 2 pretrained models with vgg16 and resNet50 architectures that are trained on the VGGface2 data set. VGGFace2 dataset is developed by researchers at the Visual Geometry Group at Oxford, includes faces across different poses and ages. The dataset contains 3.31 million images of 9131 subjects (including actors, athletes, politicians), approximately 362 images for each subject. This two pretrained models (VGG16 and ResNnet50 trained on VGGFACE2 dataset) are available in package 'keras_vggface' available (https://github.com/rcmalli/keras-vggface). this data set has a 8631 to 500 train and validation split on the identities.

Input for this two models require a face detector. For that reason, I use the detector MTCNN in package mtcnn.mtcnn. The 'preprocess_input' function of the 'keras_vggface.utils' package is used to prepare the face to become the input of the models.(required size is 224 × 224 crop of the center face image is used as input to the network). The face descriptor or the embeddings is extracted from the layer adjacent to the classifier layer. This leads to a 2048 dimensional descriptor, which is then L2 normalized.

When specifying weights: one of None (random initialization) or "vggface" (pre-training on VGGFACE datasets). this work project only uses weight pretrained on 'vggface'. This project uses the following dataset for prediction:

Selected 11 classes of celebrities in CASIA-WebFace publicly available dataset(here) This dataset is created by the Institute of Automation, Chinese Academy of Sciences (CASIA) gathered from IMDb profiles. Labels and image list of CASIA WebFace data set can be found here. CASIA WebFace Facial dataset includes 494,414 images over 10,575 identities and requires some filtering for quality. The number of images in each class averages 46 image per subject, however, for this project 11 classes with at least 180 images are selected.

  • candidate images for prediction are in 'test_dir'
In [1]:
! pip show keras-vggface
Name: keras-vggface
Version: 0.6
Summary: VGGFace implementation with Keras framework
Home-page: https://github.com/rcmalli/keras-vggface
Author: Refik Can MALLI
Author-email: mallir@itu.edu.tr
License: MIT
Location: /anaconda3/envs/DLpy37/lib/python3.7/site-packages
Requires: pyyaml, pillow, six, keras, numpy, h5py, scipy
Required-by: 
In [2]:
import keras_vggface
print(keras_vggface.__version__)
Using TensorFlow backend.
0.6
In [3]:
! python --version
Python 3.7.6
In [5]:
import os
# os.environ['KMP_DUPLICATE_LIB_OK']='True'from __future__ import absolute_import, division, print_function, unicode_literals
import pathlib
import os
import matplotlib.pyplot as plt
from matplotlib import pyplot
import numpy as np
from numpy import expand_dims
from numpy import asarray
import pandas as pd
import seaborn as sns
from IPython import display
from PIL import Image
from IPython import display
import cv2
import keras_vggface
import keras
from keras import layers
from keras import models
from keras import regularizers
from keras import optimizers
from keras.preprocessing import image
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.models import load_model
from keras.layers import *
from keras.models import Sequential
from keras_vggface.vggface import VGGFace
from keras_vggface.utils import preprocess_input
from keras_vggface.utils import decode_predictions
from keras import regularizers
from keras.utils import to_categorical
from mtcnn.mtcnn import MTCNN
In [10]:
IMG_SIZE  = 224
IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)
required_size = (IMG_SIZE, IMG_SIZE)
# Tesing set
test_dir = 'keras-facenet/data/test'
listGroupsTest = os.listdir(test_dir) 
listGroupsTest = [f for f in listGroupsTest if not f.startswith('.')]
 listGroupsTest
Out[10]:
['Teri_Hatcher',
 'Geena_Davis',
 'Neve_Campbell',
 'Kevin_Bacon',
 'Patricia_Arquette',
 'Steve_Buscemi',
 'Kim_Basinger',
 'David_Duchovny',
 'Helen_Hunt',
 'Colin_Firth',
 'Jeff_Goldblum']
In [114]:
required_size = (IMG_SIZE, IMG_SIZE)
In [99]:
def extract_face(img_path, required_size=required_size):
    # load image from file
    img = cv2.imread(img_path)
    img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    # create detector, using default weights
    detector = MTCNN()
    # detect faces in the image
    results = detector.detect_faces(img_rgb)
    # extract the bounding box associated to larger area
    bigger_face = max(results, key=lambda b: b['box'][2] * b['box'][3])
    x1, y1, width, height = bigger_face['box']
    x2, y2 = x1 + width, y1 + height
    # extract the face
    face = img_rgb[y1:y2, x1:x2]
    image = Image.fromarray(face)
    image = image.resize(required_size)
    face_array = asarray(image)
    return face_array
In [71]:
# example of face extraction:
img_path = 'Data/train/Colin_Firth/001.jpg'
img = cv2.imread(img_path)
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
plt.imshow(img_rgb)
plt.show()

face = extract_face(img_path)
plt.imshow(face)
plt.show()

Predict using pretrained models:

In [11]:
def prepare_image_pretrained(img_path, img_size = IMG_SIZE):
    face = extract_face(img_path)
    face = face.astype('float32')
    face = expand_dims(face, axis=0)
    samples = keras_vggface.utils.preprocess_input(face, version=2)
    return samples


def faceClassifier_pretrained(model, imagePath):
    preprocessed_image = prepare_image_pretrained(imagePath)
    # Use pretrained model to classify the image
    predictions = model.predict(preprocessed_image)
    results = decode_predictions(predictions)  
    return results
In [12]:
def faceRecognition_pretrained(model, test_dir):    
    predictions_list=[]
    for celeb in os.listdir(test_dir):
        if not celeb.startswith('.'):
            test_img_path = os.path.join(test_dir, celeb)
            if not test_img_path.startswith('.'):
                for test_img  in os.listdir(test_img_path):
                    if not test_img.startswith('.'):
                        fullimagepath = os.path.join(test_img_path, test_img)
                        face = prepare_image_pretrained(fullimagepath)
                        predictions = model.predict(face)
                        predictions_list.append(predictions)
                predictions = np.sum(predictions_list, axis=0)
                top_pred = decode_predictions(predictions) 
            print('Test picture from: ', celeb,':\n Prediction: ' ,top_pred, '\n')

VGG16

In [13]:
# Lets predict using exisiting pretrained vgg16 model 
vgg16 = VGGFace(model='vgg16')
faceRecognition_pretrained(vgg16, test_dir)
Test picture from:  Teri_Hatcher :
 Prediction:  [[["b'Teri_Hatcher'", 0.53289264], ["b'Marina_Sirtis'", 0.034269728], ["b'Marisol_Nichols'", 0.020355703], ["b'Bahar_Soomekh'", 0.016289664], ["b'Tamara_Feldman'", 0.014841828]]] 

Test picture from:  Geena_Davis :
 Prediction:  [[["b'Geena_Davis'", 0.97315073], ["b'Teri_Hatcher'", 0.53289354], ["b'Marina_Sirtis'", 0.034270935], ["b'Marisol_Nichols'", 0.020357871], ["b'Bahar_Soomekh'", 0.016321553]]] 

Test picture from:  Neve_Campbell :
 Prediction:  [[["b'Neve_Campbell'", 0.998247], ["b'Geena_Davis'", 0.97315073], ["b'Teri_Hatcher'", 0.53289366], ["b'Marina_Sirtis'", 0.03427095], ["b'Marisol_Nichols'", 0.02035795]]] 

Test picture from:  Kevin_Bacon :
 Prediction:  [[["b'Neve_Campbell'", 0.9982632], ["b'Geena_Davis'", 0.9731526], ["b'Kevin_Bacon'", 0.62608254], ["b'Teri_Hatcher'", 0.5328978], ["b'Marina_Sirtis'", 0.034274735]]] 

Test picture from:  Patricia_Arquette :
 Prediction:  [[["b'Neve_Campbell'", 0.998648], ["b'Geena_Davis'", 0.97316146], ["b'Kevin_Bacon'", 0.6260855], ["b'Teri_Hatcher'", 0.5328991], ["b'Olga_Kurylenko'", 0.1379144]]] 

Test picture from:  Steve_Buscemi :
 Prediction:  [[["b'Neve_Campbell'", 0.9986786], ["b'Geena_Davis'", 0.97320145], ["b'Kevin_Bacon'", 0.62609726], ["b'Teri_Hatcher'", 0.53297365], ["b'Olga_Kurylenko'", 0.13807096]]] 

Test picture from:  Kim_Basinger :
 Prediction:  [[["b'Neve_Campbell'", 0.99881387], ["b'Geena_Davis'", 0.97394097], ["b'Kevin_Bacon'", 0.62615347], ["b'Teri_Hatcher'", 0.53309536], ["b'Olga_Kurylenko'", 0.1380752]]] 

Test picture from:  David_Duchovny :
 Prediction:  [[["b'Neve_Campbell'", 0.99882376], ["b'Geena_Davis'", 0.9739497], ["b'Kevin_Bacon'", 0.62617624], ["b'Teri_Hatcher'", 0.53311515], ["b'Olga_Kurylenko'", 0.13810477]]] 

Test picture from:  Helen_Hunt :
 Prediction:  [[["b'Neve_Campbell'", 0.99882376], ["b'Helen_Hunt'", 0.99269545], ["b'Geena_Davis'", 0.9739497], ["b'Kevin_Bacon'", 0.62617624], ["b'Teri_Hatcher'", 0.53311515]]] 

Test picture from:  Colin_Firth :
 Prediction:  [[["b'Neve_Campbell'", 0.9988239], ["b'Helen_Hunt'", 0.9927002], ["b'Geena_Davis'", 0.9739514], ["b'Colin_Firth'", 0.8744785], ["b'Kevin_Bacon'", 0.62617725]]] 

Test picture from:  Jeff_Goldblum :
 Prediction:  [[["b'Neve_Campbell'", 0.99882394], ["b'Helen_Hunt'", 0.9927017], ["b'Geena_Davis'", 0.9739515], ["b'Jeff_Goldblum'", 0.97236156], ["b'Colin_Firth'", 0.8744903]]] 

In [16]:
vgg16.summary()
Model: "vggface_vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_4 (InputLayer)         (None, 224, 224, 3)       0         
_________________________________________________________________
conv1_1 (Conv2D)             (None, 224, 224, 64)      1792      
_________________________________________________________________
conv1_2 (Conv2D)             (None, 224, 224, 64)      36928     
_________________________________________________________________
pool1 (MaxPooling2D)         (None, 112, 112, 64)      0         
_________________________________________________________________
conv2_1 (Conv2D)             (None, 112, 112, 128)     73856     
_________________________________________________________________
conv2_2 (Conv2D)             (None, 112, 112, 128)     147584    
_________________________________________________________________
pool2 (MaxPooling2D)         (None, 56, 56, 128)       0         
_________________________________________________________________
conv3_1 (Conv2D)             (None, 56, 56, 256)       295168    
_________________________________________________________________
conv3_2 (Conv2D)             (None, 56, 56, 256)       590080    
_________________________________________________________________
conv3_3 (Conv2D)             (None, 56, 56, 256)       590080    
_________________________________________________________________
pool3 (MaxPooling2D)         (None, 28, 28, 256)       0         
_________________________________________________________________
conv4_1 (Conv2D)             (None, 28, 28, 512)       1180160   
_________________________________________________________________
conv4_2 (Conv2D)             (None, 28, 28, 512)       2359808   
_________________________________________________________________
conv4_3 (Conv2D)             (None, 28, 28, 512)       2359808   
_________________________________________________________________
pool4 (MaxPooling2D)         (None, 14, 14, 512)       0         
_________________________________________________________________
conv5_1 (Conv2D)             (None, 14, 14, 512)       2359808   
_________________________________________________________________
conv5_2 (Conv2D)             (None, 14, 14, 512)       2359808   
_________________________________________________________________
conv5_3 (Conv2D)             (None, 14, 14, 512)       2359808   
_________________________________________________________________
pool5 (MaxPooling2D)         (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc6 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc6/relu (Activation)        (None, 4096)              0         
_________________________________________________________________
fc7 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
fc7/relu (Activation)        (None, 4096)              0         
_________________________________________________________________
fc8 (Dense)                  (None, 2622)              10742334  
_________________________________________________________________
fc8/softmax (Activation)     (None, 2622)              0         
=================================================================
Total params: 145,002,878
Trainable params: 145,002,878
Non-trainable params: 0
_________________________________________________________________

ResNet50

In [14]:
# Lets predict using exisiting pretrained resnet50 model 
resnet50 = VGGFace(model='resnet50')
faceRecognition_pretrained(resnet50, test_dir)
Test picture from:  Teri_Hatcher :
 Prediction:  [[["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17874312], ["b' Peggy_Lipton'", 0.08712374], ["b' Valeria_Solarino'", 0.028685542], ["b' Micaela_Ramazzotti'", 0.027750503], ["b' Lana_Parrilla'", 0.02382494]]] 

Test picture from:  Geena_Davis :
 Prediction:  [[["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17875075], ["b' Melissa_Sagemiller'", 0.102681205], ["b' Peggy_Lipton'", 0.08717244], ["b' Stefanie_Stappenbeck'", 0.03472344], ["b' Dedee_Pfeiffer'", 0.032279067]]] 

Test picture from:  Neve_Campbell :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20582676], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17900482], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14541118], ["b' Natasza_Urba\\xc5\\x84ska'", 0.10859329], ["b' Melissa_Sagemiller'", 0.10268208]]] 

Test picture from:  Kevin_Bacon :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20582817], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17900893], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14541325], ["b' David_Caruso'", 0.118176855], ["b' Natasza_Urba\\xc5\\x84ska'", 0.10859391]]] 

Test picture from:  Patricia_Arquette :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20583415], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17958523], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14685358], ["b' Robin_Wright'", 0.14611211], ["b' David_Caruso'", 0.11818524]]] 

Test picture from:  Steve_Buscemi :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20584065], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17975567], ["b' Max_Pezzali'", 0.17402261], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14685449], ["b' Robin_Wright'", 0.14611967]]] 

Test picture from:  Kim_Basinger :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20584588], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17989142], ["b' Max_Pezzali'", 0.17403221], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14687073], ["b' Robin_Wright'", 0.14624599]]] 

Test picture from:  David_Duchovny :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20585334], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17991792], ["b' Max_Pezzali'", 0.17404984], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14687909], ["b' Robin_Wright'", 0.14625056]]] 

Test picture from:  Helen_Hunt :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20590375], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17995432], ["b' Max_Pezzali'", 0.17405117], ["b' Robin_Wright'", 0.16926996], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14692284]]] 

Test picture from:  Colin_Firth :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.2060756], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.17996891], ["b' Max_Pezzali'", 0.17414661], ["b' Robin_Wright'", 0.16948111], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14692947]]] 

Test picture from:  Jeff_Goldblum :
 Prediction:  [[["b' Raquel_S\\xc3\\xa1nchez-Silva'", 0.20612243], ["b' Ay\\xc5\\x9feg\\xc3\\xbcl_Aldin\\xc3\\xa7'", 0.18008852], ["b' Max_Pezzali'", 0.17415121], ["b' Robin_Wright'", 0.16948879], ["b' Ana_Claudia_Talanc\\xc3\\xb3n'", 0.14693046]]] 

In [15]:
resnet50.summary()
Model: "vggface_resnet50"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_38 (InputLayer)           (None, 224, 224, 3)  0                                            
__________________________________________________________________________________________________
conv1/7x7_s2 (Conv2D)           (None, 112, 112, 64) 9408        input_38[0][0]                   
__________________________________________________________________________________________________
conv1/7x7_s2/bn (BatchNormaliza (None, 112, 112, 64) 256         conv1/7x7_s2[0][0]               
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 112, 112, 64) 0           conv1/7x7_s2/bn[0][0]            
__________________________________________________________________________________________________
max_pooling2d_73 (MaxPooling2D) (None, 55, 55, 64)   0           activation_1[0][0]               
__________________________________________________________________________________________________
conv2_1_1x1_reduce (Conv2D)     (None, 55, 55, 64)   4096        max_pooling2d_73[0][0]           
__________________________________________________________________________________________________
conv2_1_1x1_reduce/bn (BatchNor (None, 55, 55, 64)   256         conv2_1_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 55, 55, 64)   0           conv2_1_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv2_1_3x3 (Conv2D)            (None, 55, 55, 64)   36864       activation_2[0][0]               
__________________________________________________________________________________________________
conv2_1_3x3/bn (BatchNormalizat (None, 55, 55, 64)   256         conv2_1_3x3[0][0]                
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 55, 55, 64)   0           conv2_1_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv2_1_1x1_increase (Conv2D)   (None, 55, 55, 256)  16384       activation_3[0][0]               
__________________________________________________________________________________________________
conv2_1_1x1_proj (Conv2D)       (None, 55, 55, 256)  16384       max_pooling2d_73[0][0]           
__________________________________________________________________________________________________
conv2_1_1x1_increase/bn (BatchN (None, 55, 55, 256)  1024        conv2_1_1x1_increase[0][0]       
__________________________________________________________________________________________________
conv2_1_1x1_proj/bn (BatchNorma (None, 55, 55, 256)  1024        conv2_1_1x1_proj[0][0]           
__________________________________________________________________________________________________
add_1 (Add)                     (None, 55, 55, 256)  0           conv2_1_1x1_increase/bn[0][0]    
                                                                 conv2_1_1x1_proj/bn[0][0]        
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 55, 55, 256)  0           add_1[0][0]                      
__________________________________________________________________________________________________
conv2_2_1x1_reduce (Conv2D)     (None, 55, 55, 64)   16384       activation_4[0][0]               
__________________________________________________________________________________________________
conv2_2_1x1_reduce/bn (BatchNor (None, 55, 55, 64)   256         conv2_2_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 55, 55, 64)   0           conv2_2_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv2_2_3x3 (Conv2D)            (None, 55, 55, 64)   36864       activation_5[0][0]               
__________________________________________________________________________________________________
conv2_2_3x3/bn (BatchNormalizat (None, 55, 55, 64)   256         conv2_2_3x3[0][0]                
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 55, 55, 64)   0           conv2_2_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv2_2_1x1_increase (Conv2D)   (None, 55, 55, 256)  16384       activation_6[0][0]               
__________________________________________________________________________________________________
conv2_2_1x1_increase/bn (BatchN (None, 55, 55, 256)  1024        conv2_2_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_2 (Add)                     (None, 55, 55, 256)  0           conv2_2_1x1_increase/bn[0][0]    
                                                                 activation_4[0][0]               
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 55, 55, 256)  0           add_2[0][0]                      
__________________________________________________________________________________________________
conv2_3_1x1_reduce (Conv2D)     (None, 55, 55, 64)   16384       activation_7[0][0]               
__________________________________________________________________________________________________
conv2_3_1x1_reduce/bn (BatchNor (None, 55, 55, 64)   256         conv2_3_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 55, 55, 64)   0           conv2_3_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv2_3_3x3 (Conv2D)            (None, 55, 55, 64)   36864       activation_8[0][0]               
__________________________________________________________________________________________________
conv2_3_3x3/bn (BatchNormalizat (None, 55, 55, 64)   256         conv2_3_3x3[0][0]                
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 55, 55, 64)   0           conv2_3_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv2_3_1x1_increase (Conv2D)   (None, 55, 55, 256)  16384       activation_9[0][0]               
__________________________________________________________________________________________________
conv2_3_1x1_increase/bn (BatchN (None, 55, 55, 256)  1024        conv2_3_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_3 (Add)                     (None, 55, 55, 256)  0           conv2_3_1x1_increase/bn[0][0]    
                                                                 activation_7[0][0]               
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 55, 55, 256)  0           add_3[0][0]                      
__________________________________________________________________________________________________
conv3_1_1x1_reduce (Conv2D)     (None, 28, 28, 128)  32768       activation_10[0][0]              
__________________________________________________________________________________________________
conv3_1_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_1_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 28, 28, 128)  0           conv3_1_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv3_1_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_11[0][0]              
__________________________________________________________________________________________________
conv3_1_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_1_3x3[0][0]                
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 28, 28, 128)  0           conv3_1_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv3_1_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_12[0][0]              
__________________________________________________________________________________________________
conv3_1_1x1_proj (Conv2D)       (None, 28, 28, 512)  131072      activation_10[0][0]              
__________________________________________________________________________________________________
conv3_1_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_1_1x1_increase[0][0]       
__________________________________________________________________________________________________
conv3_1_1x1_proj/bn (BatchNorma (None, 28, 28, 512)  2048        conv3_1_1x1_proj[0][0]           
__________________________________________________________________________________________________
add_4 (Add)                     (None, 28, 28, 512)  0           conv3_1_1x1_increase/bn[0][0]    
                                                                 conv3_1_1x1_proj/bn[0][0]        
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 28, 28, 512)  0           add_4[0][0]                      
__________________________________________________________________________________________________
conv3_2_1x1_reduce (Conv2D)     (None, 28, 28, 128)  65536       activation_13[0][0]              
__________________________________________________________________________________________________
conv3_2_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_2_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 28, 28, 128)  0           conv3_2_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv3_2_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_14[0][0]              
__________________________________________________________________________________________________
conv3_2_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_2_3x3[0][0]                
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 28, 28, 128)  0           conv3_2_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv3_2_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_15[0][0]              
__________________________________________________________________________________________________
conv3_2_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_2_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_5 (Add)                     (None, 28, 28, 512)  0           conv3_2_1x1_increase/bn[0][0]    
                                                                 activation_13[0][0]              
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 28, 28, 512)  0           add_5[0][0]                      
__________________________________________________________________________________________________
conv3_3_1x1_reduce (Conv2D)     (None, 28, 28, 128)  65536       activation_16[0][0]              
__________________________________________________________________________________________________
conv3_3_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_3_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_17 (Activation)      (None, 28, 28, 128)  0           conv3_3_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv3_3_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_17[0][0]              
__________________________________________________________________________________________________
conv3_3_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_3_3x3[0][0]                
__________________________________________________________________________________________________
activation_18 (Activation)      (None, 28, 28, 128)  0           conv3_3_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv3_3_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_18[0][0]              
__________________________________________________________________________________________________
conv3_3_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_3_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_6 (Add)                     (None, 28, 28, 512)  0           conv3_3_1x1_increase/bn[0][0]    
                                                                 activation_16[0][0]              
__________________________________________________________________________________________________
activation_19 (Activation)      (None, 28, 28, 512)  0           add_6[0][0]                      
__________________________________________________________________________________________________
conv3_4_1x1_reduce (Conv2D)     (None, 28, 28, 128)  65536       activation_19[0][0]              
__________________________________________________________________________________________________
conv3_4_1x1_reduce/bn (BatchNor (None, 28, 28, 128)  512         conv3_4_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_20 (Activation)      (None, 28, 28, 128)  0           conv3_4_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv3_4_3x3 (Conv2D)            (None, 28, 28, 128)  147456      activation_20[0][0]              
__________________________________________________________________________________________________
conv3_4_3x3/bn (BatchNormalizat (None, 28, 28, 128)  512         conv3_4_3x3[0][0]                
__________________________________________________________________________________________________
activation_21 (Activation)      (None, 28, 28, 128)  0           conv3_4_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv3_4_1x1_increase (Conv2D)   (None, 28, 28, 512)  65536       activation_21[0][0]              
__________________________________________________________________________________________________
conv3_4_1x1_increase/bn (BatchN (None, 28, 28, 512)  2048        conv3_4_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_7 (Add)                     (None, 28, 28, 512)  0           conv3_4_1x1_increase/bn[0][0]    
                                                                 activation_19[0][0]              
__________________________________________________________________________________________________
activation_22 (Activation)      (None, 28, 28, 512)  0           add_7[0][0]                      
__________________________________________________________________________________________________
conv4_1_1x1_reduce (Conv2D)     (None, 14, 14, 256)  131072      activation_22[0][0]              
__________________________________________________________________________________________________
conv4_1_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_1_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_23 (Activation)      (None, 14, 14, 256)  0           conv4_1_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv4_1_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_23[0][0]              
__________________________________________________________________________________________________
conv4_1_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_1_3x3[0][0]                
__________________________________________________________________________________________________
activation_24 (Activation)      (None, 14, 14, 256)  0           conv4_1_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv4_1_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_24[0][0]              
__________________________________________________________________________________________________
conv4_1_1x1_proj (Conv2D)       (None, 14, 14, 1024) 524288      activation_22[0][0]              
__________________________________________________________________________________________________
conv4_1_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_1_1x1_increase[0][0]       
__________________________________________________________________________________________________
conv4_1_1x1_proj/bn (BatchNorma (None, 14, 14, 1024) 4096        conv4_1_1x1_proj[0][0]           
__________________________________________________________________________________________________
add_8 (Add)                     (None, 14, 14, 1024) 0           conv4_1_1x1_increase/bn[0][0]    
                                                                 conv4_1_1x1_proj/bn[0][0]        
__________________________________________________________________________________________________
activation_25 (Activation)      (None, 14, 14, 1024) 0           add_8[0][0]                      
__________________________________________________________________________________________________
conv4_2_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_25[0][0]              
__________________________________________________________________________________________________
conv4_2_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_2_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_26 (Activation)      (None, 14, 14, 256)  0           conv4_2_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv4_2_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_26[0][0]              
__________________________________________________________________________________________________
conv4_2_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_2_3x3[0][0]                
__________________________________________________________________________________________________
activation_27 (Activation)      (None, 14, 14, 256)  0           conv4_2_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv4_2_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_27[0][0]              
__________________________________________________________________________________________________
conv4_2_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_2_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_9 (Add)                     (None, 14, 14, 1024) 0           conv4_2_1x1_increase/bn[0][0]    
                                                                 activation_25[0][0]              
__________________________________________________________________________________________________
activation_28 (Activation)      (None, 14, 14, 1024) 0           add_9[0][0]                      
__________________________________________________________________________________________________
conv4_3_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_28[0][0]              
__________________________________________________________________________________________________
conv4_3_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_3_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_29 (Activation)      (None, 14, 14, 256)  0           conv4_3_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv4_3_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_29[0][0]              
__________________________________________________________________________________________________
conv4_3_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_3_3x3[0][0]                
__________________________________________________________________________________________________
activation_30 (Activation)      (None, 14, 14, 256)  0           conv4_3_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv4_3_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_30[0][0]              
__________________________________________________________________________________________________
conv4_3_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_3_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_10 (Add)                    (None, 14, 14, 1024) 0           conv4_3_1x1_increase/bn[0][0]    
                                                                 activation_28[0][0]              
__________________________________________________________________________________________________
activation_31 (Activation)      (None, 14, 14, 1024) 0           add_10[0][0]                     
__________________________________________________________________________________________________
conv4_4_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_31[0][0]              
__________________________________________________________________________________________________
conv4_4_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_4_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_32 (Activation)      (None, 14, 14, 256)  0           conv4_4_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv4_4_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_32[0][0]              
__________________________________________________________________________________________________
conv4_4_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_4_3x3[0][0]                
__________________________________________________________________________________________________
activation_33 (Activation)      (None, 14, 14, 256)  0           conv4_4_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv4_4_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_33[0][0]              
__________________________________________________________________________________________________
conv4_4_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_4_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_11 (Add)                    (None, 14, 14, 1024) 0           conv4_4_1x1_increase/bn[0][0]    
                                                                 activation_31[0][0]              
__________________________________________________________________________________________________
activation_34 (Activation)      (None, 14, 14, 1024) 0           add_11[0][0]                     
__________________________________________________________________________________________________
conv4_5_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_34[0][0]              
__________________________________________________________________________________________________
conv4_5_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_5_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_35 (Activation)      (None, 14, 14, 256)  0           conv4_5_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv4_5_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_35[0][0]              
__________________________________________________________________________________________________
conv4_5_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_5_3x3[0][0]                
__________________________________________________________________________________________________
activation_36 (Activation)      (None, 14, 14, 256)  0           conv4_5_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv4_5_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_36[0][0]              
__________________________________________________________________________________________________
conv4_5_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_5_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_12 (Add)                    (None, 14, 14, 1024) 0           conv4_5_1x1_increase/bn[0][0]    
                                                                 activation_34[0][0]              
__________________________________________________________________________________________________
activation_37 (Activation)      (None, 14, 14, 1024) 0           add_12[0][0]                     
__________________________________________________________________________________________________
conv4_6_1x1_reduce (Conv2D)     (None, 14, 14, 256)  262144      activation_37[0][0]              
__________________________________________________________________________________________________
conv4_6_1x1_reduce/bn (BatchNor (None, 14, 14, 256)  1024        conv4_6_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_38 (Activation)      (None, 14, 14, 256)  0           conv4_6_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv4_6_3x3 (Conv2D)            (None, 14, 14, 256)  589824      activation_38[0][0]              
__________________________________________________________________________________________________
conv4_6_3x3/bn (BatchNormalizat (None, 14, 14, 256)  1024        conv4_6_3x3[0][0]                
__________________________________________________________________________________________________
activation_39 (Activation)      (None, 14, 14, 256)  0           conv4_6_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv4_6_1x1_increase (Conv2D)   (None, 14, 14, 1024) 262144      activation_39[0][0]              
__________________________________________________________________________________________________
conv4_6_1x1_increase/bn (BatchN (None, 14, 14, 1024) 4096        conv4_6_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_13 (Add)                    (None, 14, 14, 1024) 0           conv4_6_1x1_increase/bn[0][0]    
                                                                 activation_37[0][0]              
__________________________________________________________________________________________________
activation_40 (Activation)      (None, 14, 14, 1024) 0           add_13[0][0]                     
__________________________________________________________________________________________________
conv5_1_1x1_reduce (Conv2D)     (None, 7, 7, 512)    524288      activation_40[0][0]              
__________________________________________________________________________________________________
conv5_1_1x1_reduce/bn (BatchNor (None, 7, 7, 512)    2048        conv5_1_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_41 (Activation)      (None, 7, 7, 512)    0           conv5_1_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv5_1_3x3 (Conv2D)            (None, 7, 7, 512)    2359296     activation_41[0][0]              
__________________________________________________________________________________________________
conv5_1_3x3/bn (BatchNormalizat (None, 7, 7, 512)    2048        conv5_1_3x3[0][0]                
__________________________________________________________________________________________________
activation_42 (Activation)      (None, 7, 7, 512)    0           conv5_1_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv5_1_1x1_increase (Conv2D)   (None, 7, 7, 2048)   1048576     activation_42[0][0]              
__________________________________________________________________________________________________
conv5_1_1x1_proj (Conv2D)       (None, 7, 7, 2048)   2097152     activation_40[0][0]              
__________________________________________________________________________________________________
conv5_1_1x1_increase/bn (BatchN (None, 7, 7, 2048)   8192        conv5_1_1x1_increase[0][0]       
__________________________________________________________________________________________________
conv5_1_1x1_proj/bn (BatchNorma (None, 7, 7, 2048)   8192        conv5_1_1x1_proj[0][0]           
__________________________________________________________________________________________________
add_14 (Add)                    (None, 7, 7, 2048)   0           conv5_1_1x1_increase/bn[0][0]    
                                                                 conv5_1_1x1_proj/bn[0][0]        
__________________________________________________________________________________________________
activation_43 (Activation)      (None, 7, 7, 2048)   0           add_14[0][0]                     
__________________________________________________________________________________________________
conv5_2_1x1_reduce (Conv2D)     (None, 7, 7, 512)    1048576     activation_43[0][0]              
__________________________________________________________________________________________________
conv5_2_1x1_reduce/bn (BatchNor (None, 7, 7, 512)    2048        conv5_2_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_44 (Activation)      (None, 7, 7, 512)    0           conv5_2_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv5_2_3x3 (Conv2D)            (None, 7, 7, 512)    2359296     activation_44[0][0]              
__________________________________________________________________________________________________
conv5_2_3x3/bn (BatchNormalizat (None, 7, 7, 512)    2048        conv5_2_3x3[0][0]                
__________________________________________________________________________________________________
activation_45 (Activation)      (None, 7, 7, 512)    0           conv5_2_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv5_2_1x1_increase (Conv2D)   (None, 7, 7, 2048)   1048576     activation_45[0][0]              
__________________________________________________________________________________________________
conv5_2_1x1_increase/bn (BatchN (None, 7, 7, 2048)   8192        conv5_2_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_15 (Add)                    (None, 7, 7, 2048)   0           conv5_2_1x1_increase/bn[0][0]    
                                                                 activation_43[0][0]              
__________________________________________________________________________________________________
activation_46 (Activation)      (None, 7, 7, 2048)   0           add_15[0][0]                     
__________________________________________________________________________________________________
conv5_3_1x1_reduce (Conv2D)     (None, 7, 7, 512)    1048576     activation_46[0][0]              
__________________________________________________________________________________________________
conv5_3_1x1_reduce/bn (BatchNor (None, 7, 7, 512)    2048        conv5_3_1x1_reduce[0][0]         
__________________________________________________________________________________________________
activation_47 (Activation)      (None, 7, 7, 512)    0           conv5_3_1x1_reduce/bn[0][0]      
__________________________________________________________________________________________________
conv5_3_3x3 (Conv2D)            (None, 7, 7, 512)    2359296     activation_47[0][0]              
__________________________________________________________________________________________________
conv5_3_3x3/bn (BatchNormalizat (None, 7, 7, 512)    2048        conv5_3_3x3[0][0]                
__________________________________________________________________________________________________
activation_48 (Activation)      (None, 7, 7, 512)    0           conv5_3_3x3/bn[0][0]             
__________________________________________________________________________________________________
conv5_3_1x1_increase (Conv2D)   (None, 7, 7, 2048)   1048576     activation_48[0][0]              
__________________________________________________________________________________________________
conv5_3_1x1_increase/bn (BatchN (None, 7, 7, 2048)   8192        conv5_3_1x1_increase[0][0]       
__________________________________________________________________________________________________
add_16 (Add)                    (None, 7, 7, 2048)   0           conv5_3_1x1_increase/bn[0][0]    
                                                                 activation_46[0][0]              
__________________________________________________________________________________________________
activation_49 (Activation)      (None, 7, 7, 2048)   0           add_16[0][0]                     
__________________________________________________________________________________________________
avg_pool (AveragePooling2D)     (None, 1, 1, 2048)   0           activation_49[0][0]              
__________________________________________________________________________________________________
flatten_25 (Flatten)            (None, 2048)         0           avg_pool[0][0]                   
__________________________________________________________________________________________________
classifier (Dense)              (None, 8631)         17684919    flatten_25[0][0]                 
==================================================================================================
Total params: 41,246,071
Trainable params: 41,192,951
Non-trainable params: 53,120
__________________________________________________________________________________________________

Report:

Pretrained models performance:

  • Although pretrained models include the classes asssociated to selected subjects, they do not do a good job in predicting them.

  • VGG15 predicts only 3 our of the 11 pictures in the tes_dir, where as the Resnet50 None of them.

2. How to improve performance of existing pretrained models using deep learning techniques?

Some explanation on the namings:

-conv_base is used to expleain that it respresents the convolutional base  and does not have the top layer

-FE used to specify the object is used in the Feature Extraction 

-TL used to specify the object is used in the Transfer Learning 

-FT used to specify the object is used in the Fine Tuning 

Feature Extract (FE):

Extract features and labels in train, validation, and test folders by predicting from the convolutional base of the pretrained models Created Models using Extracted Features from Convolutional Base of VGG16 and ResNet50 Reshape them to be inputted into a model with dense layers Use dropouts and regularizes for regularization to avoid overfitting.

In [101]:
# create a plot function to avoid rewriting:
In [17]:
def plot_history(history, title, clr1='blue', clr2='green'):
  # helper function to plot the loss and accuracy 
    plt.figure(figsize=(18, 7))
    acc = history.history['accuracy']
    val_acc = history.history['val_accuracy']
    loss = history.history['loss']
    val_loss = history.history['val_loss']
    
    plt.suptitle('Categorical Plotting')

    plt.subplot(1, 2, 1)
    plt.plot(range(1, len(acc)+1), acc, color=clr1, label='Training Accuracy ' )
    plt.plot(range(1, len(acc)+1), val_acc, color=clr2, linestyle='dashed', label='Validation Accuracy' )
    plt.legend()
    plt.ylabel('Accuracy')
    plt.ylim([min(plt.ylim()),1])
    plt.title('Training and Validation Accuracy')

    plt.subplot(1, 2, 2)
    plt.plot(range(1, len(acc)+1), loss, color=clr1,  label='Training Loss ')
    plt.plot(range(1, len(acc)+1), val_loss, color=clr2, linestyle='dashed', label='Validation Loss')
    plt.legend()
    plt.ylabel('Loss')
    plt.title('Training and Validation Loss')
    plt.xlabel('epoch')
    plt.suptitle(title)
    plt.show()

Loading the convolutional base of model VGG16 for FE:

In [108]:
conv_base_vgg16_FE = VGGFace(input_shape=IMG_SHAPE,model='vgg16',weights='vggface',include_top=False)
In [110]:
conv_base_vgg16_FE.summary()
Model: "vggface_vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_288 (InputLayer)       (None, 224, 224, 3)       0         
_________________________________________________________________
conv1_1 (Conv2D)             (None, 224, 224, 64)      1792      
_________________________________________________________________
conv1_2 (Conv2D)             (None, 224, 224, 64)      36928     
_________________________________________________________________
pool1 (MaxPooling2D)         (None, 112, 112, 64)      0         
_________________________________________________________________
conv2_1 (Conv2D)             (None, 112, 112, 128)     73856     
_________________________________________________________________
conv2_2 (Conv2D)             (None, 112, 112, 128)     147584    
_________________________________________________________________
pool2 (MaxPooling2D)         (None, 56, 56, 128)       0         
_________________________________________________________________
conv3_1 (Conv2D)             (None, 56, 56, 256)       295168    
_________________________________________________________________
conv3_2 (Conv2D)             (None, 56, 56, 256)       590080    
_________________________________________________________________
conv3_3 (Conv2D)             (None, 56, 56, 256)       590080    
_________________________________________________________________
pool3 (MaxPooling2D)         (None, 28, 28, 256)       0         
_________________________________________________________________
conv4_1 (Conv2D)             (None, 28, 28, 512)       1180160   
_________________________________________________________________
conv4_2 (Conv2D)             (None, 28, 28, 512)       2359808   
_________________________________________________________________
conv4_3 (Conv2D)             (None, 28, 28, 512)       2359808   
_________________________________________________________________
pool4 (MaxPooling2D)         (None, 14, 14, 512)       0         
_________________________________________________________________
conv5_1 (Conv2D)             (None, 14, 14, 512)       2359808   
_________________________________________________________________
conv5_2 (Conv2D)             (None, 14, 14, 512)       2359808   
_________________________________________________________________
conv5_3 (Conv2D)             (None, 14, 14, 512)       2359808   
_________________________________________________________________
pool5 (MaxPooling2D)         (None, 7, 7, 512)         0         
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________

Extract features and labels: for train validation and test, and reshape them to get prepared for the required input dimension for a model with dense layers: (input_dim=7 7 512)

In [104]:
base_dir = 'Data'
test_dir = 'keras-facenet/data/test'
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')

datagen = ImageDataGenerator(rescale=1./255)
batch_size = 20

def extract_features_vgg16(directory, sample_count):
    features = np.zeros(shape=(sample_count, 7, 7, 512))
    labels = np.zeros(shape=(sample_count))
    generator = datagen.flow_from_directory(
        directory,
        target_size=(IMG_SIZE, IMG_SIZE),
        batch_size=batch_size,
        class_mode='sparse')
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base_vgg16_FE.predict(inputs_batch)
        features[i * batch_size : (i + 1) * batch_size] = features_batch
        labels[i * batch_size : (i + 1) * batch_size] = labels_batch
        i += 1

        if i * batch_size >= sample_count:
            # Note that since generators yield data indefinitely in a loop,
            # we must `break` after every image has been seen once.
            break
    return features, labels

#  extract the features 
train_features_vgg16, train_labels_vgg16 = extract_features_vgg16(train_dir, 2142)
validation_features_vgg16, validation_labels_vgg16 = extract_features_vgg16(validation_dir, 1089)
test_features_vgg16, test_labels_vgg16 = extract_features_vgg16(test_dir, 11)


train_features_vgg16 = np.reshape(train_features_vgg16, (2142, 7*7* 512))
validation_features_vgg16= np.reshape(validation_features_vgg16, (1089, 7* 7* 512))
test_features_vgg16 = np.reshape(test_features_vgg16, (11, 7* 7* 512))
Found 2142 images belonging to 11 classes.
Found 1089 images belonging to 11 classes.
Found 11 images belonging to 11 classes.
In [20]:
vgg16_FE = keras.models.Sequential()
vgg16_FE.add(keras.layers.Dense(256, activation='relu', input_dim=7 * 7 * 512))
vgg16_FE.add(keras.layers.Dropout(0.5))
vgg16_FE.add(keras.layers.Dense(128, activation='relu'))
vgg16_FE.add(keras.layers.Dropout(0.5))
vgg16_FE.add(keras.layers.Dense(11, activation='softmax'))

#compile model
vgg16_FE.compile(optimizer=keras.optimizers.Adam(lr=2e-2),
              loss='sparse_categorical_crossentropy',metrics=['accuracy'])

#  fit model using 20 epochs
history_vgg16_FE= vgg16_FE.fit(train_features_vgg16, train_labels_vgg16,
                    epochs=20,
                    batch_size=20,
                    validation_data=(validation_features_vgg16, validation_labels_vgg16))
Train on 2142 samples, validate on 1089 samples
Epoch 1/20
2142/2142 [==============================] - 8s 4ms/step - loss: 3.7787 - accuracy: 0.9916 - val_loss: 2.8267 - val_accuracy: 0.9826
Epoch 2/20
2142/2142 [==============================] - 7s 3ms/step - loss: 1.4461 - accuracy: 0.9907 - val_loss: 3.1294 - val_accuracy: 0.9844
Epoch 3/20
2142/2142 [==============================] - 7s 3ms/step - loss: 3.6255 - accuracy: 0.9855 - val_loss: 1.6346 - val_accuracy: 0.9881
Epoch 4/20
2142/2142 [==============================] - 7s 3ms/step - loss: 3.8657 - accuracy: 0.9911 - val_loss: 3.0933 - val_accuracy: 0.9826
Epoch 5/20
2142/2142 [==============================] - 7s 3ms/step - loss: 1.9335 - accuracy: 0.9921 - val_loss: 0.5237 - val_accuracy: 0.9835
Epoch 6/20
2142/2142 [==============================] - 7s 3ms/step - loss: 1.1917 - accuracy: 0.9916 - val_loss: 1.3274 - val_accuracy: 0.9826
Epoch 7/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.9990 - accuracy: 0.9925 - val_loss: 0.2368 - val_accuracy: 0.9835
Epoch 8/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.6873 - accuracy: 0.9925 - val_loss: 1.7667 - val_accuracy: 0.9844
Epoch 9/20
2142/2142 [==============================] - 7s 3ms/step - loss: 1.3253 - accuracy: 0.9911 - val_loss: 0.0702 - val_accuracy: 0.9853
Epoch 10/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.3056 - accuracy: 0.9916 - val_loss: 0.0489 - val_accuracy: 0.9844
Epoch 11/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.2871 - accuracy: 0.9911 - val_loss: 0.0455 - val_accuracy: 0.9844
Epoch 12/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.1270 - accuracy: 0.9916 - val_loss: 0.0464 - val_accuracy: 0.9844
Epoch 13/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.6959 - accuracy: 0.9925 - val_loss: 0.0461 - val_accuracy: 0.9844
Epoch 14/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.1848 - accuracy: 0.9921 - val_loss: 0.0456 - val_accuracy: 0.9844
Epoch 15/20
2142/2142 [==============================] - 8s 4ms/step - loss: 0.3103 - accuracy: 0.9916 - val_loss: 0.0454 - val_accuracy: 0.9844
Epoch 16/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.1776 - accuracy: 0.9925 - val_loss: 0.0444 - val_accuracy: 0.9844
Epoch 17/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.2464 - accuracy: 0.9921 - val_loss: 0.0436 - val_accuracy: 0.9844
Epoch 18/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.5468 - accuracy: 0.9935 - val_loss: 0.0427 - val_accuracy: 0.9844
Epoch 19/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.5977 - accuracy: 0.9921 - val_loss: 0.0430 - val_accuracy: 0.9844
Epoch 20/20
2142/2142 [==============================] - 7s 3ms/step - loss: 0.0509 - accuracy: 0.9921 - val_loss: 0.0432 - val_accuracy: 0.9844
In [21]:
vgg16_FE.save('vgg16_FE.h5')
In [22]:
plot_history(history_vgg16_FE, 'Feature Extract - created model from extracted features of vgg16 convolutional base')

Loading the convolutional base of model resnet50 for FE

In [24]:
conv_base_resnet50_FE = VGGFace(input_shape=IMG_SHAPE,model='resnet50',weights='vggface',include_top=False)
In [105]:
base_dir = 'Data'
test_dir = 'keras-facenet/data/test'
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'val')

datagen = ImageDataGenerator(rescale=1./255)
batch_size = 20

def extract_features_resnet50(directory, sample_count):
    features = np.zeros(shape=(sample_count, 1, 1, 2048))
    labels = np.zeros(shape=(sample_count))
    
#     Data Augmentation
    generator = datagen.flow_from_directory(
        directory,
        target_size=(IMG_SIZE, IMG_SIZE),
        batch_size=batch_size,
        class_mode='sparse')
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base_resnet50_FE.predict(inputs_batch)
        features[i * batch_size : (i + 1) * batch_size] = features_batch
                
        labels[i * batch_size : (i + 1) * batch_size] = labels_batch
        i += 1
#         print(inputs_batch, labels_batch )
        break
        if i * batch_size >= sample_count:
            # Note that since generators yield data indefinitely in a loop,
            # we must `break` after every image has been seen once.
            break
    return features, labels

train_features_resnet50, train_labels_resnet50 = extract_features_resnet50(train_dir, 2142)
validation_features_resnet50, validation_labels_resnet50 = extract_features_resnet50(validation_dir, 1089)
test_features_resnet50, test_labels_resnet50 = extract_features_resnet50(test_dir, 11)
Found 2142 images belonging to 11 classes.
Found 1089 images belonging to 11 classes.
Found 11 images belonging to 11 classes.
In [28]:
train_features_resnet50 = np.reshape(train_features_resnet50, (2142,  1 * 1 * 2048))
validation_features_resnet50 = np.reshape(validation_features_resnet50, (1089, 1 * 1 * 2048))
test_features_resnet50 = np.reshape(test_features_resnet50, (11, 1 * 1 * 2048))
In [29]:
resnet50_FE = keras.models.Sequential()
resnet50_FE.add(keras.layers.Dense(512, kernel_regularizer=regularizers.l2(lamb), activation='relu', input_dim=1 * 1 * 2048))
resnet50_FE.add(keras.layers.Dropout(0.5))
resnet50_FE.add(keras.layers.Dense(128, kernel_regularizer=regularizers.l2(lamb), activation='relu'))
resnet50_FE.add(keras.layers.Dropout(0.5))
resnet50_FE.add(keras.layers.Dense(11, activation='softmax'))

resnet50_FE.compile(optimizer=keras.optimizers.Adam(lr=2e-2),
              loss='sparse_categorical_crossentropy',metrics=['accuracy'])

history_resnet50_FE = resnet50_FE.fit(train_features_resnet50, train_labels_resnet50,
                    epochs=20,
                    batch_size=20,
                    validation_data=(validation_features_resnet50, validation_labels_resnet50))
Train on 2142 samples, validate on 1089 samples
Epoch 1/20
2142/2142 [==============================] - 2s 867us/step - loss: 1.6240 - accuracy: 0.9916 - val_loss: 0.7360 - val_accuracy: 0.9862
Epoch 2/20
2142/2142 [==============================] - 2s 801us/step - loss: 0.4993 - accuracy: 0.9921 - val_loss: 0.3757 - val_accuracy: 0.9826
Epoch 3/20
2142/2142 [==============================] - 2s 784us/step - loss: 0.2959 - accuracy: 0.9921 - val_loss: 0.1765 - val_accuracy: 0.9862
Epoch 4/20
2142/2142 [==============================] - 2s 770us/step - loss: 0.1888 - accuracy: 0.9916 - val_loss: 0.2327 - val_accuracy: 0.9862
Epoch 5/20
2142/2142 [==============================] - 2s 774us/step - loss: 0.2349 - accuracy: 0.9921 - val_loss: 0.4435 - val_accuracy: 0.9862
Epoch 6/20
2142/2142 [==============================] - 2s 784us/step - loss: 0.5025 - accuracy: 0.9911 - val_loss: 0.3661 - val_accuracy: 0.9862
Epoch 7/20
2142/2142 [==============================] - 2s 784us/step - loss: 0.3191 - accuracy: 0.9935 - val_loss: 0.2323 - val_accuracy: 0.9862
Epoch 8/20
2142/2142 [==============================] - 2s 797us/step - loss: 0.1312 - accuracy: 0.9921 - val_loss: 0.1025 - val_accuracy: 0.9862
Epoch 9/20
2142/2142 [==============================] - 2s 795us/step - loss: 0.1685 - accuracy: 0.9921 - val_loss: 0.1953 - val_accuracy: 0.9862
Epoch 10/20
2142/2142 [==============================] - 2s 761us/step - loss: 0.1525 - accuracy: 0.9921 - val_loss: 0.3834 - val_accuracy: 0.9862
Epoch 11/20
2142/2142 [==============================] - 2s 757us/step - loss: 0.1436 - accuracy: 0.9921 - val_loss: 0.4764 - val_accuracy: 0.9862
Epoch 12/20
2142/2142 [==============================] - 2s 747us/step - loss: 0.2285 - accuracy: 0.9916 - val_loss: 0.2487 - val_accuracy: 0.9862
Epoch 13/20
2142/2142 [==============================] - 2s 757us/step - loss: 0.1657 - accuracy: 0.9921 - val_loss: 0.1373 - val_accuracy: 0.9862
Epoch 14/20
2142/2142 [==============================] - 2s 785us/step - loss: 0.0962 - accuracy: 0.9921 - val_loss: 0.0895 - val_accuracy: 0.9862
Epoch 15/20
2142/2142 [==============================] - 1s 685us/step - loss: 0.0826 - accuracy: 0.9921 - val_loss: 0.0876 - val_accuracy: 0.9862
Epoch 16/20
2142/2142 [==============================] - 2s 759us/step - loss: 0.0929 - accuracy: 0.9921 - val_loss: 0.0788 - val_accuracy: 0.9862
Epoch 17/20
2142/2142 [==============================] - 2s 775us/step - loss: 0.0982 - accuracy: 0.9921 - val_loss: 0.2898 - val_accuracy: 0.9862
Epoch 18/20
2142/2142 [==============================] - 2s 781us/step - loss: 0.1725 - accuracy: 0.9921 - val_loss: 0.0977 - val_accuracy: 0.9862
Epoch 19/20
2142/2142 [==============================] - 2s 776us/step - loss: 0.0799 - accuracy: 0.9921 - val_loss: 0.1137 - val_accuracy: 0.9862
Epoch 20/20
2142/2142 [==============================] - 2s 768us/step - loss: 0.0646 - accuracy: 0.9921 - val_loss: 0.1199 - val_accuracy: 0.9862
In [30]:
resnet50_FE.summary()
Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_165 (Dense)            (None, 512)               1049088   
_________________________________________________________________
dropout_3 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_166 (Dense)            (None, 128)               65664     
_________________________________________________________________
dropout_4 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_167 (Dense)            (None, 11)                1419      
=================================================================
Total params: 1,116,171
Trainable params: 1,116,171
Non-trainable params: 0
_________________________________________________________________
In [31]:
plot_history(history_resnet50_FE, 'Feature Extract - created model from extracted features of resnet50 ')
In [33]:
resnet50_FE.save('resnet50_FE_30epochs.h5')

Report:

  • The accuracy of training and test remains almost constant through different epochs. Also the loss of training and validation becomes almost equal in 20 epochs. No need for training with more epochs.

  • The validation accuracy remains less that training, but they are close.

  • 20 epochs seems to be a good choice, accuracy has been constant and loss values are decreasing across both training and validation.

Transfer Learning (TL)

Build a model on top of the convolutional base of the VGG16 and ResNet50 Freeze the parameters of the convolutional base Train parameters in added dense and classification layers Use dropouts and regularizes for regularization to avoid overfitting

Data Augmentation

  • Images are augmented based on horizontal flip, 45 rotation, and zooming for training. The validation images are not augmented.
In [37]:
pathFolder = './Data/'
IMG_SIZE  = 224
IMG_SHAPE = (IMG_SIZE, IMG_SIZE, 3)

#  image augmentation

# We are getting our train_generator, validation_generator and test_generator ready :

train_datagen=ImageDataGenerator(preprocessing_function=keras_vggface.utils.preprocess_input,
                                                  horizontal_flip=True, 
                                                  rotation_range=45, 
                                                  zoom_range=[0.8,1.0]) 

test_datagen=ImageDataGenerator(preprocessing_function=keras_vggface.utils.preprocess_input) 


# Training set
pathTrain = pathFolder + 'train/'
listGroupsTrain = os.listdir(pathTrain) # the directory path
listGroupsTrain = [f for f in listGroupsTrain if not f.startswith('.')]


# Validation set
pathVal = pathFolder + 'val/'
listGroupsValid = os.listdir(pathVal) # the directory path
listGroupsValid = [f for f in listGroupsValid if not f.startswith('.')]

# Tesing set
test_dir = 'keras-facenet/data/test'
listGroupsTest = os.listdir(test_dir) 
listGroupsTest = [f for f in listGroupsTest if not f.startswith('.')]


# Load the data into the ImageDataGenerator
train_generator=train_datagen.flow_from_directory(pathFolder+'train',
                                                 target_size=(224,224),
                                                 color_mode='rgb',
                                                 batch_size=64,
                                                 class_mode='categorical',
                                                 shuffle=True, 
                                                 classes=listGroupsTrain)
                                                 
validation_generator=test_datagen.flow_from_directory(pathFolder+'val',
                                                 target_size=(224,224),
                                                 color_mode='rgb',
                                                 batch_size=64,
                                                 class_mode='categorical',
                                                 shuffle=False, 
                                                 classes=listGroupsValid)

test_generator = test_datagen.flow_from_directory(test_dir,
                                                    target_size=(224,224),
                                                  color_mode='rgb',
                                                    batch_size=64,
                                                    class_mode='categorical')
Found 2142 images belonging to 11 classes.
Found 1089 images belonging to 11 classes.
Found 11 images belonging to 11 classes.

Steps Transfer learning :

- Get the convolutional base from the pretrained model
- Freeze the layers in the convolutional base
- Add dense layers and the classifiction layer to adjust added layer's weight by retrain model to my dataset's categories
-  In one model, 1 hidden layer is added, in the next model, 2 hidden layers are added along with the classifcation layer.

Transfer Learning on ResNet50 :

Added 1 dense and a softmax layer and retrained the added layers:

  • Having conv base layers frozen.
In [38]:
conv_base_resnet50 = VGGFace(input_shape=IMG_SHAPE,model='resnet50',
                             weights='vggface',include_top=False)
conv_base_resnet50.trainable = False


# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
resnet50_TL_1layer = Sequential([
    conv_base_resnet50,                       
    keras.layers.Flatten(),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
        kernel_regularizer=regularizers.l1(kernel_weight), 
        bias_regularizer=regularizers.l1(bias_weight)),
    keras.layers.Dense(11, activation='softmax', name='FC_2',
         kernel_regularizer=regularizers.l1(kernel_weight), 
         bias_regularizer=regularizers.l1(bias_weight))])

# compile
resnet50_TL_1layer.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
                     loss='categorical_crossentropy' ,metrics=['accuracy'])
# fit the model using similar hyperparameters as before
step_size_train=train_generator.n//train_generator.batch_size  
history_resnet50_TL_1layer= resnet50_TL_1layer.fit_generator(generator=train_generator,
                                       validation_data=validation_generator,
                                       steps_per_epoch=step_size_train,
                                       epochs=20)

resnet50_TL_1layer.save('resnet50_TL_1layer.h5')
resnet50_TL_1layer.save_weights('resnet50_TL_1layer_weights.h5')
plot_history(history_resnet50_TL_1layer, 'Transfer learning  - Resnet50 - re-trained added 1 hidden dense and output layers')
Epoch 1/20
33/33 [==============================] - 216s 7s/step - loss: 514.1299 - accuracy: 0.1179 - val_loss: 514.5323 - val_accuracy: 0.0735
Epoch 2/20
33/33 [==============================] - 223s 7s/step - loss: 507.2674 - accuracy: 0.1684 - val_loss: 507.7248 - val_accuracy: 0.1028
Epoch 3/20
33/33 [==============================] - 214s 6s/step - loss: 500.9229 - accuracy: 0.2411 - val_loss: 501.2010 - val_accuracy: 0.1635
Epoch 4/20
33/33 [==============================] - 215s 7s/step - loss: 494.6628 - accuracy: 0.3168 - val_loss: 494.5741 - val_accuracy: 0.2397
Epoch 5/20
33/33 [==============================] - 212s 6s/step - loss: 488.3743 - accuracy: 0.3752 - val_loss: 488.1242 - val_accuracy: 0.3177
Epoch 6/20
33/33 [==============================] - 212s 6s/step - loss: 482.2300 - accuracy: 0.4437 - val_loss: 481.6822 - val_accuracy: 0.3875
Epoch 7/20
33/33 [==============================] - 214s 6s/step - loss: 476.0472 - accuracy: 0.4750 - val_loss: 475.2384 - val_accuracy: 0.4490
Epoch 8/20
33/33 [==============================] - 213s 6s/step - loss: 469.8951 - accuracy: 0.5091 - val_loss: 468.9101 - val_accuracy: 0.5115
Epoch 9/20
33/33 [==============================] - 214s 6s/step - loss: 463.7847 - accuracy: 0.5582 - val_loss: 462.4962 - val_accuracy: 0.5427
Epoch 10/20
33/33 [==============================] - 214s 6s/step - loss: 457.6659 - accuracy: 0.5837 - val_loss: 456.2107 - val_accuracy: 0.5840
Epoch 11/20
33/33 [==============================] - 212s 6s/step - loss: 451.5328 - accuracy: 0.6027 - val_loss: 449.9157 - val_accuracy: 0.6180
Epoch 12/20
33/33 [==============================] - 211s 6s/step - loss: 445.5337 - accuracy: 0.6169 - val_loss: 443.7289 - val_accuracy: 0.6419
Epoch 13/20
33/33 [==============================] - 213s 6s/step - loss: 439.5001 - accuracy: 0.6218 - val_loss: 437.7404 - val_accuracy: 0.6676
Epoch 14/20
33/33 [==============================] - 211s 6s/step - loss: 433.4752 - accuracy: 0.6665 - val_loss: 431.6091 - val_accuracy: 0.6823
Epoch 15/20
33/33 [==============================] - 212s 6s/step - loss: 427.4972 - accuracy: 0.6814 - val_loss: 425.3814 - val_accuracy: 0.6951
Epoch 16/20
33/33 [==============================] - 213s 6s/step - loss: 421.5750 - accuracy: 0.6925 - val_loss: 419.3508 - val_accuracy: 0.7163
Epoch 17/20
33/33 [==============================] - 214s 6s/step - loss: 415.6876 - accuracy: 0.7045 - val_loss: 413.4669 - val_accuracy: 0.7300
Epoch 18/20
33/33 [==============================] - 211s 6s/step - loss: 409.7522 - accuracy: 0.7177 - val_loss: 407.5237 - val_accuracy: 0.7374
Epoch 19/20
33/33 [==============================] - 217s 7s/step - loss: 403.9449 - accuracy: 0.7244 - val_loss: 401.5539 - val_accuracy: 0.7484
Epoch 20/20
33/33 [==============================] - 210s 6s/step - loss: 398.1151 - accuracy: 0.7383 - val_loss: 395.7479 - val_accuracy: 0.7603
In [126]:
resnet50_TL_1layer.save('resnet50_TL_1layer.h5')
In [111]:
plot_history(history_resnet50_TL_1layer, 'Transfer learning  - Resnet50 - re-trained added 1 hidden dense and output layers')
In [125]:
FaceRecognition_testdata(resnet50_TL_1layer, listGroupsTest)

Added 2 dense and a softmax layer and retrained the added layers:

  • Having conv base layers frozen.
In [41]:
conv_base_resnet50 = VGGFace(input_shape=IMG_SHAPE,model='resnet50',
                             weights='vggface',include_top=False)
In [44]:
conv_base_resnet50.trainable = False
number_groupsTrain = len(listGroupsTrain)
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512

resnet50_TL = Sequential([
    conv_base_resnet50,                       # base from resnet50 
    keras.layers.Flatten(),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_1"),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_2",
        kernel_regularizer=regularizers.l1(kernel_weight), 
        bias_regularizer=regularizers.l1(bias_weight)),
    keras.layers.Dense(11, activation='softmax', name='FC_3',
        kernel_regularizer=regularizers.l1(kernel_weight), 
        bias_regularizer=regularizers.l1(bias_weight))])

resnet50_TL.summary()
Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
vggface_resnet50 (Model)     (None, 1, 1, 2048)        23561152  
_________________________________________________________________
flatten_71 (Flatten)         (None, 2048)              0         
_________________________________________________________________
FC_1 (Dense)                 (None, 512)               1049088   
_________________________________________________________________
FC_2 (Dense)                 (None, 512)               262656    
_________________________________________________________________
FC_3 (Dense)                 (None, 11)                5643      
=================================================================
Total params: 24,878,539
Trainable params: 1,317,387
Non-trainable params: 23,561,152
_________________________________________________________________
In [45]:
resnet50_TL.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
                     loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size  
history_resnet50_TL = resnet50_TL.fit_generator(generator=train_generator,
                                       validation_data=validation_generator,
                                       steps_per_epoch=step_size_train,
                                       epochs=20)
Epoch 1/20
33/33 [==============================] - 215s 7s/step - loss: 207.9778 - accuracy: 0.1631 - val_loss: 208.5120 - val_accuracy: 0.3085
Epoch 2/20
33/33 [==============================] - 214s 6s/step - loss: 205.5557 - accuracy: 0.3797 - val_loss: 206.1152 - val_accuracy: 0.5308
Epoch 3/20
33/33 [==============================] - 210s 6s/step - loss: 203.5203 - accuracy: 0.5739 - val_loss: 203.9610 - val_accuracy: 0.6841
Epoch 4/20
33/33 [==============================] - 211s 6s/step - loss: 201.5967 - accuracy: 0.6954 - val_loss: 201.9017 - val_accuracy: 0.7778
Epoch 5/20
33/33 [==============================] - 213s 6s/step - loss: 199.7438 - accuracy: 0.7661 - val_loss: 199.7233 - val_accuracy: 0.8411
Epoch 6/20
33/33 [==============================] - 213s 6s/step - loss: 197.9002 - accuracy: 0.8253 - val_loss: 197.6738 - val_accuracy: 0.8788
Epoch 7/20
33/33 [==============================] - 212s 6s/step - loss: 196.1231 - accuracy: 0.8600 - val_loss: 195.7300 - val_accuracy: 0.8962
Epoch 8/20
33/33 [==============================] - 211s 6s/step - loss: 194.3791 - accuracy: 0.8797 - val_loss: 193.9606 - val_accuracy: 0.9118
Epoch 9/20
33/33 [==============================] - 210s 6s/step - loss: 192.6610 - accuracy: 0.8929 - val_loss: 192.0019 - val_accuracy: 0.9238
Epoch 10/20
33/33 [==============================] - 214s 6s/step - loss: 190.9391 - accuracy: 0.9066 - val_loss: 190.2163 - val_accuracy: 0.9302
Epoch 11/20
33/33 [==============================] - 213s 6s/step - loss: 189.2463 - accuracy: 0.9110 - val_loss: 188.4973 - val_accuracy: 0.9339
Epoch 12/20
33/33 [==============================] - 213s 6s/step - loss: 187.5873 - accuracy: 0.9158 - val_loss: 186.7834 - val_accuracy: 0.9348
Epoch 13/20
33/33 [==============================] - 214s 6s/step - loss: 185.9186 - accuracy: 0.9152 - val_loss: 185.0413 - val_accuracy: 0.9412
Epoch 14/20
33/33 [==============================] - 214s 6s/step - loss: 184.2933 - accuracy: 0.9124 - val_loss: 183.4481 - val_accuracy: 0.9421
Epoch 15/20
33/33 [==============================] - 212s 6s/step - loss: 182.6377 - accuracy: 0.9193 - val_loss: 181.7207 - val_accuracy: 0.9403
Epoch 16/20
33/33 [==============================] - 219s 7s/step - loss: 181.0302 - accuracy: 0.9124 - val_loss: 180.1216 - val_accuracy: 0.9440
Epoch 17/20
33/33 [==============================] - 212s 6s/step - loss: 179.3695 - accuracy: 0.9312 - val_loss: 178.4805 - val_accuracy: 0.9421
Epoch 18/20
33/33 [==============================] - 210s 6s/step - loss: 177.7602 - accuracy: 0.9330 - val_loss: 176.8620 - val_accuracy: 0.9440
Epoch 19/20
33/33 [==============================] - 215s 7s/step - loss: 176.1717 - accuracy: 0.9225 - val_loss: 175.1914 - val_accuracy: 0.9467
Epoch 20/20
33/33 [==============================] - 213s 6s/step - loss: 174.5491 - accuracy: 0.9304 - val_loss: 173.6574 - val_accuracy: 0.9440
In [123]:
resnet50_TL.save('resnet50_TL.h5')
resnet50_TL.save_weights('resnet50_TL_weights.h5')
In [103]:
plot_history(history_resnet50_TL, 'Transfer learning - resnet50 - re-trained added 2 hidden dense and output layers')
In [ ]:
#  Create a function that goes through all images in test_dir and predicts them using the specified model:
In [52]:
def FaceRecognition_testdata(model, listGroupsTest):
    i=0
    for celeb in listGroupsTest:
        if not celeb.startswith('.'):
            test_img_path = os.path.join(test_dir, celeb)
            if not test_img_path.startswith('.'):
                for test_img  in os.listdir(test_img_path):
                    if not test_img.startswith('.'):

                        test_identity = 'Not in database'

                        test_img_fullpath = os.path.join(test_img_path, test_img)
                        face = extract_face(test_img_fullpath)
                        face = face.astype('float32')
                        face = expand_dims(face, axis=0)
                        preprocessed_face = preprocess_input(face, version=2)

                        #normalzed the color code
    #                     preprocessed_face = face / 255
                        prediction = model.predict(preprocessed_face)

                        if np.max(prediction) > 0.5:
                            name = np.argmax(prediction, axis=-1)
                            test_identity = listGroupsTest[name[0]]
                        img = cv2.imread(os.path.join(test_img_path, test_img))
                        img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
                        plt.imshow(img_rgb)
                        plt.title(test_identity)
                        plt.show()
#                         print('test image is : ', test_identity)
In [53]:
FaceRecognition_testdata(resnet50_TL, listGroupsTest)
test image is :  Teri_Hatcher
test image is :  Geena_Davis
test image is :  Neve_Campbell
test image is :  Kevin_Bacon
test image is :  Patricia_Arquette
test image is :  Steve_Buscemi
test image is :  Kim_Basinger
test image is :  David_Duchovny
test image is :  Helen_Hunt
test image is :  Colin_Firth
test image is :  Jeff_Goldblum
In [122]:
FaceRecognition_testdata(resnet50_TL, listGroupsTest)

Transfer Learning on VGG16 :

Add 1 hidden dense and output layer on the convolutional base and retrain added layers:

  • Having conv base layers frozen.
In [106]:
conv_base_vgg = VGGFace(input_shape=IMG_SHAPE,model='vgg16',
                        weights='vggface',include_top=False)
In [60]:
conv_base_vgg.trainable = False
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512

vgg16_TL_1layer = Sequential([
    conv_base_vgg,                       
    keras.layers.Flatten(),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
        kernel_regularizer=regularizers.l1(kernel_weight), 
        bias_regularizer=regularizers.l1(bias_weight)),
    keras.layers.Dense(11, activation='softmax', name='FC_2',
         kernel_regularizer=regularizers.l1(kernel_weight), 
         bias_regularizer=regularizers.l1(bias_weight))])


vgg16_TL_1layer.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
                     loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size  
history_vgg16_TL_1layer= vgg16_TL_1layer.fit_generator(generator=train_generator,
                                       validation_data=validation_generator,
                                       steps_per_epoch=step_size_train,
                                       epochs=20)

vgg16_TL_1layer.save('vgg16_TL_1layer.h5')
vgg16_TL_1layer.save_weights('vgg16_TL_1layer_weights.h5')
plot_history(history_vgg16_TL_1layer, 'Transfer learning  - vgg16 - re-trained added 1 hidden dense and output layers')
Epoch 1/20
33/33 [==============================] - 284s 9s/step - loss: 1946.8883 - accuracy: 0.3590 - val_loss: 1917.3114 - val_accuracy: 0.7006
Epoch 2/20
33/33 [==============================] - 304s 9s/step - loss: 1886.3787 - accuracy: 0.6704 - val_loss: 1854.4717 - val_accuracy: 0.8301
Epoch 3/20
33/33 [==============================] - 312s 9s/step - loss: 1828.0357 - accuracy: 0.7608 - val_loss: 1795.4243 - val_accuracy: 0.8724
Epoch 4/20
33/33 [==============================] - 321s 10s/step - loss: 1767.4123 - accuracy: 0.8162 - val_loss: 1734.4683 - val_accuracy: 0.8972
Epoch 5/20
33/33 [==============================] - 320s 10s/step - loss: 1705.9247 - accuracy: 0.8414 - val_loss: 1672.7388 - val_accuracy: 0.8972
Epoch 6/20
33/33 [==============================] - 315s 10s/step - loss: 1643.9116 - accuracy: 0.8488 - val_loss: 1611.1769 - val_accuracy: 0.9238
Epoch 7/20
33/33 [==============================] - 318s 10s/step - loss: 1582.0651 - accuracy: 0.8705 - val_loss: 1549.4282 - val_accuracy: 0.9265
Epoch 8/20
33/33 [==============================] - 320s 10s/step - loss: 1520.9929 - accuracy: 0.8653 - val_loss: 1489.2161 - val_accuracy: 0.9311
Epoch 9/20
33/33 [==============================] - 327s 10s/step - loss: 1461.0303 - accuracy: 0.8773 - val_loss: 1429.6272 - val_accuracy: 0.9330
Epoch 10/20
33/33 [==============================] - 327s 10s/step - loss: 1401.4113 - accuracy: 0.8932 - val_loss: 1370.0436 - val_accuracy: 0.9330
Epoch 11/20
33/33 [==============================] - 334s 10s/step - loss: 1342.2476 - accuracy: 0.8963 - val_loss: 1311.3723 - val_accuracy: 0.9348
Epoch 12/20
33/33 [==============================] - 327s 10s/step - loss: 1283.8556 - accuracy: 0.9081 - val_loss: 1253.6094 - val_accuracy: 0.9421
Epoch 13/20
33/33 [==============================] - 324s 10s/step - loss: 1226.0967 - accuracy: 0.9163 - val_loss: 1196.0933 - val_accuracy: 0.9376
Epoch 14/20
33/33 [==============================] - 333s 10s/step - loss: 1169.3446 - accuracy: 0.9181 - val_loss: 1140.0630 - val_accuracy: 0.9421
Epoch 15/20
33/33 [==============================] - 329s 10s/step - loss: 1113.8201 - accuracy: 0.9211 - val_loss: 1085.2728 - val_accuracy: 0.9421
Epoch 16/20
33/33 [==============================] - 331s 10s/step - loss: 1059.6225 - accuracy: 0.9212 - val_loss: 1031.8328 - val_accuracy: 0.9458
Epoch 17/20
33/33 [==============================] - 332s 10s/step - loss: 1006.6187 - accuracy: 0.9366 - val_loss: 979.0645 - val_accuracy: 0.9477
Epoch 18/20
33/33 [==============================] - 317s 10s/step - loss: 954.3414 - accuracy: 0.9256 - val_loss: 927.8055 - val_accuracy: 0.9467
Epoch 19/20
33/33 [==============================] - 316s 10s/step - loss: 903.5670 - accuracy: 0.9346 - val_loss: 877.4529 - val_accuracy: 0.9467
Epoch 20/20
33/33 [==============================] - 321s 10s/step - loss: 854.1662 - accuracy: 0.9328 - val_loss: 829.0189 - val_accuracy: 0.9458

Add 2 hidden dense layers and output layer on the convolutional base and retrain added layers:

  • having conv base layers frozen.
In [62]:
conv_base_vgg = VGGFace(input_shape=IMG_SHAPE,model='vgg16',weights='vggface',include_top=False)
conv_base_vgg.trainable = False

# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
vgg16_TL = Sequential([
    conv_base_vgg,                       
    keras.layers.Flatten(),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
        kernel_regularizer=regularizers.l1(kernel_weight), 
        bias_regularizer=regularizers.l1(bias_weight)),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_2",
        kernel_regularizer=regularizers.l1(kernel_weight), 
        bias_regularizer=regularizers.l1(bias_weight)),
    keras.layers.Dense(11, activation='softmax', name='FC_3',
         kernel_regularizer=regularizers.l1(kernel_weight), 
         bias_regularizer=regularizers.l1(bias_weight))])
vgg16_TL.summary()
Model: "sequential_7"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
vggface_vgg16 (Model)        (None, 7, 7, 512)         14714688  
_________________________________________________________________
flatten_96 (Flatten)         (None, 25088)             0         
_________________________________________________________________
FC_1 (Dense)                 (None, 512)               12845568  
_________________________________________________________________
FC_2 (Dense)                 (None, 512)               262656    
_________________________________________________________________
FC_3 (Dense)                 (None, 11)                5643      
=================================================================
Total params: 27,828,555
Trainable params: 13,113,867
Non-trainable params: 14,714,688
_________________________________________________________________
In [64]:
vgg16_TL.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
                     loss='categorical_crossentropy' ,metrics=['accuracy'])

step_size_train=train_generator.n//train_generator.batch_size  

history_vgg16_TL= vgg16_TL.fit_generator(generator=train_generator,
                                       validation_data=validation_generator,
                                       steps_per_epoch=step_size_train,
                                       epochs=20)
Epoch 1/20
33/33 [==============================] - 325s 10s/step - loss: 2137.6640 - accuracy: 0.3373 - val_loss: 2098.7161 - val_accuracy: 0.5840
Epoch 2/20
33/33 [==============================] - 333s 10s/step - loss: 2070.4592 - accuracy: 0.6304 - val_loss: 2035.2307 - val_accuracy: 0.7429
Epoch 3/20
33/33 [==============================] - 334s 10s/step - loss: 2005.0398 - accuracy: 0.6998 - val_loss: 1969.5917 - val_accuracy: 0.8255
Epoch 4/20
33/33 [==============================] - 317s 10s/step - loss: 1938.6780 - accuracy: 0.7681 - val_loss: 1903.1599 - val_accuracy: 0.8540
Epoch 5/20
33/33 [==============================] - 324s 10s/step - loss: 1871.7089 - accuracy: 0.7973 - val_loss: 1836.0433 - val_accuracy: 0.8669
Epoch 6/20
33/33 [==============================] - 348s 11s/step - loss: 1804.1800 - accuracy: 0.8253 - val_loss: 1768.8500 - val_accuracy: 0.8880
Epoch 7/20
33/33 [==============================] - 343s 10s/step - loss: 1737.2965 - accuracy: 0.8439 - val_loss: 1702.4409 - val_accuracy: 0.9008
Epoch 8/20
33/33 [==============================] - 340s 10s/step - loss: 1670.9994 - accuracy: 0.8518 - val_loss: 1636.5992 - val_accuracy: 0.9036
Epoch 9/20
33/33 [==============================] - 356s 11s/step - loss: 1605.8298 - accuracy: 0.8556 - val_loss: 1571.9482 - val_accuracy: 0.9100
Epoch 10/20
33/33 [==============================] - 346s 10s/step - loss: 1541.1852 - accuracy: 0.8782 - val_loss: 1507.6620 - val_accuracy: 0.9164
Epoch 11/20
33/33 [==============================] - 351s 11s/step - loss: 1477.4131 - accuracy: 0.8869 - val_loss: 1444.5215 - val_accuracy: 0.9201
Epoch 12/20
33/33 [==============================] - 359s 11s/step - loss: 1414.9561 - accuracy: 0.8840 - val_loss: 1382.8218 - val_accuracy: 0.9247
Epoch 13/20
33/33 [==============================] - 356s 11s/step - loss: 1353.7862 - accuracy: 0.8912 - val_loss: 1322.3547 - val_accuracy: 0.9284
Epoch 14/20
33/33 [==============================] - 348s 11s/step - loss: 1293.8748 - accuracy: 0.9010 - val_loss: 1262.9558 - val_accuracy: 0.9284
Epoch 15/20
33/33 [==============================] - 335s 10s/step - loss: 1235.1182 - accuracy: 0.8938 - val_loss: 1205.1327 - val_accuracy: 0.9376
Epoch 16/20
33/33 [==============================] - 332s 10s/step - loss: 1177.9112 - accuracy: 0.9044 - val_loss: 1148.4297 - val_accuracy: 0.9376
Epoch 17/20
33/33 [==============================] - 324s 10s/step - loss: 1121.7083 - accuracy: 0.9172 - val_loss: 1092.9076 - val_accuracy: 0.9385
Epoch 18/20
33/33 [==============================] - 318s 10s/step - loss: 1066.9005 - accuracy: 0.9203 - val_loss: 1038.8553 - val_accuracy: 0.9385
Epoch 19/20
33/33 [==============================] - 316s 10s/step - loss: 1013.3419 - accuracy: 0.9235 - val_loss: 985.9893 - val_accuracy: 0.9394
Epoch 20/20
33/33 [==============================] - 321s 10s/step - loss: 961.2954 - accuracy: 0.9219 - val_loss: 934.8681 - val_accuracy: 0.9403
In [65]:
vgg16_TL.save('vgg16_TL.h5')
vgg16_TL.save_weights('vgg16_TL_weights.h5')
In [66]:
# vgg16_TL = load_model('vgg16_TL.h5')
FaceRecognition_testdata(vgg16_TL, listGroupsTest)
test image is :  Teri_Hatcher
test image is :  Geena_Davis
test image is :  Neve_Campbell
test image is :  Kevin_Bacon
test image is :  Colin_Firth
test image is :  Steve_Buscemi
test image is :  Kim_Basinger
test image is :  David_Duchovny
test image is :  Kim_Basinger
test image is :  Colin_Firth
test image is :  Jeff_Goldblum
In [67]:
plot_history(history_vgg16_TL, 'Transfer learning  - vgg16 - re-trained added 2 hidden dense and output layers ')

Report:

  • Adding 2 hidden dense layers versus 1 hidden dense layer significantly improved the accuracy and the performance of the model created on Resnet50's convolutional base (val acc increased from approximately 76% to approximately 94%).

  • However, in the case of models using VGG16's convolutional base, the accuracy of the performance in both cases are ~ 94%

  • Validation and training accuracy and loss change together with a slight difference. The model is neither overfitting not underfitting

  • 20 epochs seems to be a good choice since loss has been decreasing and train and validation accuracy are very close. Neither overfitting or underfitting.

Fine tunning

  • Build a model on top of the convolutional base of the VGG16 and ResNet50
  • Unfreeze the parameters of all layers in the last convolutional block
  • The remaining layers in convolutional base are frozen
  • Train parameters in the last convolutional block, added dense and classification layers
  • Use regularizes for regularization to avoid overfitting

Fine Tuning on ResNet50:

In [72]:
conv_base_resnet_FT = VGGFace(input_shape=
                                        IMG_SHAPE, model='resnet50',weights='vggface', include_top=False)

# The number of the layers in the base model
NumOfLayers = len(conv_base_resnet_FT.layers)
print("Number of layers in the base model: ", NumOfLayers)
Number of layers in the base model:  174
In [74]:
conv_base_resnet_FT.trainable = True
for layer in conv_base_resnet_FT.layers[0:141]:
    layer.trainable =  False
    i+=1
In [76]:
number_groupsTrain = len(listGroupsTrain)
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512

resnet50_FT = Sequential([
    conv_base_resnet_FT,
    keras.layers.Flatten(),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
                kernel_regularizer=regularizers.l1(kernel_weight), 
                bias_regularizer=regularizers.l1(bias_weight)),
    keras.layers.Dense(11, activation='softmax', name='FC_2',
                kernel_regularizer=regularizers.l1(kernel_weight), 
                bias_regularizer=regularizers.l1(bias_weight))
])
resnet50_FT.summary()
Model: "sequential_8"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
vggface_resnet50 (Model)     (None, 1, 1, 2048)        23561152  
_________________________________________________________________
flatten_145 (Flatten)        (None, 2048)              0         
_________________________________________________________________
FC_1 (Dense)                 (None, 512)               1049088   
_________________________________________________________________
FC_2 (Dense)                 (None, 11)                5643      
=================================================================
Total params: 24,615,883
Trainable params: 16,019,467
Non-trainable params: 8,596,416
_________________________________________________________________
In [77]:
resnet50_FT.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
                     loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size
history_resnet50_FT= resnet50_FT.fit_generator(generator=train_generator,
                                                  validation_data=validation_generator,
                                                  steps_per_epoch=step_size_train,
                                                  epochs=20)
Epoch 1/20
33/33 [==============================] - 251s 8s/step - loss: 511.0422 - accuracy: 0.5746 - val_loss: 507.4561 - val_accuracy: 0.8118
Epoch 2/20
33/33 [==============================] - 272s 8s/step - loss: 503.1316 - accuracy: 0.8561 - val_loss: 499.3997 - val_accuracy: 0.8999
Epoch 3/20
33/33 [==============================] - 284s 9s/step - loss: 496.2203 - accuracy: 0.9148 - val_loss: 492.4884 - val_accuracy: 0.9284
Epoch 4/20
33/33 [==============================] - 279s 8s/step - loss: 489.4308 - accuracy: 0.9283 - val_loss: 485.7546 - val_accuracy: 0.9403
Epoch 5/20
33/33 [==============================] - 271s 8s/step - loss: 482.6761 - accuracy: 0.9504 - val_loss: 479.0760 - val_accuracy: 0.9541
Epoch 6/20
33/33 [==============================] - 271s 8s/step - loss: 476.0086 - accuracy: 0.9490 - val_loss: 472.4304 - val_accuracy: 0.9587
Epoch 7/20
33/33 [==============================] - 260s 8s/step - loss: 469.3557 - accuracy: 0.9615 - val_loss: 465.8300 - val_accuracy: 0.9614
Epoch 8/20
33/33 [==============================] - 259s 8s/step - loss: 462.7541 - accuracy: 0.9629 - val_loss: 459.2705 - val_accuracy: 0.9633
Epoch 9/20
33/33 [==============================] - 259s 8s/step - loss: 456.1943 - accuracy: 0.9716 - val_loss: 452.7515 - val_accuracy: 0.9642
Epoch 10/20
33/33 [==============================] - 260s 8s/step - loss: 449.6893 - accuracy: 0.9735 - val_loss: 446.2768 - val_accuracy: 0.9642
Epoch 11/20
33/33 [==============================] - 260s 8s/step - loss: 443.2233 - accuracy: 0.9783 - val_loss: 439.8398 - val_accuracy: 0.9642
Epoch 12/20
33/33 [==============================] - 269s 8s/step - loss: 436.8331 - accuracy: 0.9740 - val_loss: 433.4659 - val_accuracy: 0.9660
Epoch 13/20
33/33 [==============================] - 269s 8s/step - loss: 430.4518 - accuracy: 0.9788 - val_loss: 427.1341 - val_accuracy: 0.9660
Epoch 14/20
33/33 [==============================] - 270s 8s/step - loss: 424.1313 - accuracy: 0.9808 - val_loss: 420.8484 - val_accuracy: 0.9642
Epoch 15/20
33/33 [==============================] - 274s 8s/step - loss: 417.8602 - accuracy: 0.9822 - val_loss: 414.6006 - val_accuracy: 0.9651
Epoch 16/20
33/33 [==============================] - 276s 8s/step - loss: 411.6172 - accuracy: 0.9877 - val_loss: 408.3956 - val_accuracy: 0.9669
Epoch 17/20
33/33 [==============================] - 264s 8s/step - loss: 405.4384 - accuracy: 0.9858 - val_loss: 402.2314 - val_accuracy: 0.9688
Epoch 18/20
33/33 [==============================] - 275s 8s/step - loss: 399.3059 - accuracy: 0.9844 - val_loss: 396.1147 - val_accuracy: 0.9706
Epoch 19/20
33/33 [==============================] - 286s 9s/step - loss: 393.2093 - accuracy: 0.9870 - val_loss: 390.0548 - val_accuracy: 0.9697
Epoch 20/20
33/33 [==============================] - 283s 9s/step - loss: 387.1705 - accuracy: 0.9856 - val_loss: 384.0705 - val_accuracy: 0.9706
In [78]:
resnet50_FT.save('resnet50_FT.h5')
resnet50_FT.save_weights('resnet50_FT_weights.h5')
In [79]:
plot_history(history_resnet50_FT, 'Fine tuning - resnet50 - last convolutional block (conv5) and added dense layers re-trained')
In [80]:
FaceRecognition_testdata(resnet50_FT, listGroupsTest)
test image is :  Teri_Hatcher
test image is :  Geena_Davis
test image is :  Neve_Campbell
test image is :  Kevin_Bacon
test image is :  Patricia_Arquette
test image is :  Steve_Buscemi
test image is :  Kim_Basinger
test image is :  David_Duchovny
test image is :  Helen_Hunt
test image is :  Colin_Firth
test image is :  Jeff_Goldblum

Fine Tuning on VGG16:

In [107]:
# Fine-tune only the last convolution
conv_base_vgg_FT = VGGFace(input_shape=
                                        IMG_SHAPE, model='vgg16',weights='vggface', include_top=False)
In [88]:
# The number of the layers in the base model
NumOfLayers = len(conv_base_vgg_FT.layers)
print("Number of layers in the base model: ", NumOfLayers)
Number of layers in the base model:  19
In [90]:
number_groupsTrain = len(listGroupsTrain)
# Regularize the dense layer by using L1
kernel_weight = 0.02
bias_weight = 0.02
hidden_dim =512
vgg16_FT = Sequential([
      conv_base_vgg_FT,
      keras.layers.Flatten(),
    keras.layers.Dense(hidden_dim, activation='relu', name="FC_1",
                    kernel_regularizer=regularizers.l1(kernel_weight), 
                    bias_regularizer=regularizers.l1(bias_weight)),
 keras.layers.Dense(11, activation='softmax', name='FC_2',#)])
                    kernel_regularizer=regularizers.l1(kernel_weight), 
                    bias_regularizer=regularizers.l1(bias_weight))])
vgg16_FT.summary()
Model: "sequential_10"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
vggface_vgg16 (Model)        (None, 7, 7, 512)         14714688  
_________________________________________________________________
flatten_169 (Flatten)        (None, 25088)             0         
_________________________________________________________________
FC_1 (Dense)                 (None, 512)               12845568  
_________________________________________________________________
FC_2 (Dense)                 (None, 11)                5643      
=================================================================
Total params: 27,565,899
Trainable params: 27,565,899
Non-trainable params: 0
_________________________________________________________________
In [91]:
conv_base_vgg_FT.trainable = True
for layer in conv_base_vgg_FT.layers[0:15]:
    layer.trainable =  False
    i+=1
In [92]:
for layer in conv_base_vgg_FT.layers:
    print(layer, layer.trainable)
<keras.engine.input_layer.InputLayer object at 0x1ae413e7d0> False
<keras.layers.convolutional.Conv2D object at 0x1ae413e910> False
<keras.layers.convolutional.Conv2D object at 0x1ae7898290> False
<keras.layers.pooling.MaxPooling2D object at 0x1ae413ec10> False
<keras.layers.convolutional.Conv2D object at 0x1ae20cd0d0> False
<keras.layers.convolutional.Conv2D object at 0x1ae4145ad0> False
<keras.layers.pooling.MaxPooling2D object at 0x1ae212aed0> False
<keras.layers.convolutional.Conv2D object at 0x1ae20b8ed0> False
<keras.layers.convolutional.Conv2D object at 0x1ae414af90> False
<keras.layers.convolutional.Conv2D object at 0x1ae4150f10> False
<keras.layers.pooling.MaxPooling2D object at 0x1ae4156590> False
<keras.layers.convolutional.Conv2D object at 0x1ae415c590> False
<keras.layers.convolutional.Conv2D object at 0x1ae4161710> False
<keras.layers.convolutional.Conv2D object at 0x1ae4167e90> False
<keras.layers.pooling.MaxPooling2D object at 0x1ae41679d0> False
<keras.layers.convolutional.Conv2D object at 0x1ae416a9d0> True
<keras.layers.convolutional.Conv2D object at 0x1ae416fb50> True
<keras.layers.convolutional.Conv2D object at 0x1ae4175a10> True
<keras.layers.pooling.MaxPooling2D object at 0x1ae4175a50> True
In [93]:
vgg16_FT.compile(optimizer=keras.optimizers.RMSprop(lr=1e-5),
                     loss='categorical_crossentropy' ,metrics=['accuracy'])
step_size_train=train_generator.n//train_generator.batch_size
history_vgg16_FT= vgg16_FT.fit_generator(generator=train_generator,
                                                  validation_data=validation_generator,
                                                  steps_per_epoch=step_size_train,
                                                  epochs=20)
Epoch 1/20
33/33 [==============================] - 345s 10s/step - loss: 1941.9436 - accuracy: 0.4346 - val_loss: 1904.2771 - val_accuracy: 0.7355
Epoch 2/20
33/33 [==============================] - 371s 11s/step - loss: 1876.6668 - accuracy: 0.7026 - val_loss: 1842.2394 - val_accuracy: 0.8356
Epoch 3/20
33/33 [==============================] - 395s 12s/step - loss: 1811.9191 - accuracy: 0.7873 - val_loss: 1776.3794 - val_accuracy: 0.8733
Epoch 4/20
33/33 [==============================] - 399s 12s/step - loss: 1744.4469 - accuracy: 0.8426 - val_loss: 1708.4407 - val_accuracy: 0.8898
Epoch 5/20
33/33 [==============================] - 395s 12s/step - loss: 1676.2930 - accuracy: 0.8460 - val_loss: 1640.5051 - val_accuracy: 0.9036
Epoch 6/20
33/33 [==============================] - 401s 12s/step - loss: 1608.2639 - accuracy: 0.8624 - val_loss: 1572.8936 - val_accuracy: 0.9118
Epoch 7/20
33/33 [==============================] - 400s 12s/step - loss: 1540.6393 - accuracy: 0.8898 - val_loss: 1505.3125 - val_accuracy: 0.9183
Epoch 8/20
33/33 [==============================] - 364s 11s/step - loss: 1473.2330 - accuracy: 0.8980 - val_loss: 1438.2783 - val_accuracy: 0.9192
Epoch 9/20
33/33 [==============================] - 347s 11s/step - loss: 1406.5259 - accuracy: 0.9167 - val_loss: 1372.0951 - val_accuracy: 0.9256
Epoch 10/20
33/33 [==============================] - 353s 11s/step - loss: 1340.8773 - accuracy: 0.9216 - val_loss: 1307.0358 - val_accuracy: 0.9403
Epoch 11/20
33/33 [==============================] - 373s 11s/step - loss: 1276.3934 - accuracy: 0.9148 - val_loss: 1243.3884 - val_accuracy: 0.9431
Epoch 12/20
33/33 [==============================] - 367s 11s/step - loss: 1213.1756 - accuracy: 0.9365 - val_loss: 1180.7964 - val_accuracy: 0.9440
Epoch 13/20
33/33 [==============================] - 367s 11s/step - loss: 1151.3768 - accuracy: 0.9312 - val_loss: 1119.7837 - val_accuracy: 0.9477
Epoch 14/20
33/33 [==============================] - 363s 11s/step - loss: 1090.7925 - accuracy: 0.9398 - val_loss: 1059.8130 - val_accuracy: 0.9495
Epoch 15/20
33/33 [==============================] - 368s 11s/step - loss: 1031.6050 - accuracy: 0.9490 - val_loss: 1001.4525 - val_accuracy: 0.9421
Epoch 16/20
33/33 [==============================] - 378s 11s/step - loss: 973.9151 - accuracy: 0.9471 - val_loss: 944.6382 - val_accuracy: 0.9504
Epoch 17/20
33/33 [==============================] - 366s 11s/step - loss: 917.8763 - accuracy: 0.9514 - val_loss: 889.5390 - val_accuracy: 0.9449
Epoch 18/20
33/33 [==============================] - 365s 11s/step - loss: 863.5979 - accuracy: 0.9538 - val_loss: 835.9622 - val_accuracy: 0.9440
Epoch 19/20
33/33 [==============================] - 356s 11s/step - loss: 810.7809 - accuracy: 0.9560 - val_loss: 784.0436 - val_accuracy: 0.9550
Epoch 20/20
33/33 [==============================] - 360s 11s/step - loss: 759.6740 - accuracy: 0.9557 - val_loss: 733.7641 - val_accuracy: 0.9568
In [94]:
vgg16_FT.save('vgg16_FT.h5')
vgg16_FT.save_weights('vgg16_FT_weights.h5')
In [96]:
plot_history(history_vgg16_FT, 'Fine tuning - vgg16 - last convolutional block (conv5) and added Dense layers re-trained')
In [97]:
FaceRecognition_testdata(vgg16_FT, listGroupsTest)
test image is :  Teri_Hatcher
test image is :  Geena_Davis
test image is :  Neve_Campbell
test image is :  Kevin_Bacon
test image is :  Patricia_Arquette
test image is :  Steve_Buscemi
test image is :  Kim_Basinger
test image is :  David_Duchovny
test image is :  Helen_Hunt
test image is :  Colin_Firth
test image is :  Jeff_Goldblum

Report:

  • Finetuning improved the performance accuracies more than transfer learning in models using both architectures

  • Model using VGG16 convolutional base responded better than ReNet50 to the transfer learning regardless of the 1 or 2 layers added ( with accuracy ~93%. val-acc ~94%). Model using ReNet50 convolutional base only compensated to the same accuracy when 2 hidden dense layers were added

  • Model using Resnet50 convolutional base responded to finetuning as both the training and validation accuracy increased to approximately 98%.

Highlight and Explanation of Reports:

  1. Repurposing existing models, using a moderate size dataset, to match an extracted face to a known individual can significantly improve the prediction accuracy of pretrained models.
  2. The performance of repurposed models strongly depends on the underlying network architecture and not necessarily the depth or number of layers. VGG16 yields higher validation accuracy with less layers retrained. VGG16 and Resnet50 models are both pretrained on the same VGGFace2 dataset. However, when more of the layers are retrained, model using ResNet50 improved more.
  3. Fine tuning improved the validation accuracies more than transfer learning in models created VGG16 and ResNet50’s convolutional base.
  4. Models using VGG16’s convolutional base responded better than ReNet50 to the transfer learning regardless of the 1 or 2 layers added. Models using ReNet50’s convolutional base only compensated to the same accuracy when 2 hidden dense layers were added.
  5. Model using Resnet50’s convolutional base responded better than VGG16 to finetuning. (model using VGG16 convolutional base improved from ~93% in TL to ~ 95% in FT, while using Resnet50 improved from ~93% in TL to ~97% in FT)
  6. Models in Transfer learning and finetuning provided a better fit than models created using extracted features. In transfer learning and fine tuning sections, training and validation accuracy improved together and remain very close. Models are neither overfitting nor underfitting. On the other hand, in the models created using extracted features, the validation accuracy remain less training accuracy.
  7. The performance of repurposed models strongly depends on the number of images in the underlying datasets of pretrained models, as FaceNet with 1 million identities strongly over performs VGG16 and ResNet50 with 9,131 identities.
  8. The initial dataset used to train them and not necessarily the depth or number of layers.